A Data Stream Management System for Network Traffic Management

The problem of obtaining the best possible network performance in the growing Internet has given rise to the need for efficient network traffic management [2]. Broadly, traffic management can be divided into three tasks: (1) collecting data, e.g., network topology and utilization data, router configuration data; (2) processing the collected data, e.g., to detect problems such as link failure, to determine the best ways to optimize network performance; (3) deploying mechanisms for controlling network traffic. We focus on tasks (1) and (2) in the context of a large Internet Service Provider (ISP) such as AT&T or Sprint. Data collected to enable traffic management applications in an ISP includes: network packet and flow traces; active measurements of packet delay, loss, and throughput; router forwarding tables and configuration data; and Simple Network Management Protocol (SNMP) data maintained by various network elements (e.g., routers, switches) [2, 5]. Since networks need to be running all the time, much of this data is collected continuously (although on different time scales) and results in very large and fast-growing databases. For example, packet traces collected in the Sprint IP backbone amount to 600 Gigabytes of data per day [5]. Different kinds of processing must be performed on the collected data to enable sophisticated traffic management applications. The network topology is maintained by joining SNMP data and/or configuration data from different network elements [2]. Statistics of link and router utilization are maintained by aggregating packet traces [5] or from SNMP data [2]. Packet losses, per-hop and end-to-end delays, and network throughput are measured either by joining packet traces collected from multiple points in the network [5], or by using a dedicated system that generates network traffic to